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ABSTRACT 



Several recent studies of Naval Communications have 
determined that the system is not operating properly. This 
paper presents a test procedure which will improve 
the performance of the equipment that composes the 
Communications System. The need for a test procedure , modes 
of failure and present test procedures are examined. It is 
determined that present test procedures referred to as system 
level tests are mislabelled and are actually siabsystems level 
tests. A Systems Level Test, the Standard Measurement Tech- 
nique (SMT) which applies inputs of known degradation to a 
system and quantifies the output is presented. Present 
problems and potential applications of SMT are discussed. 
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I. 



COMMUNICATIONS AVAILABILITY AND THE NEED FOR A SYSTEM 
TEST PROCEDURE 



In recent years several studies of the Naval Communica- 
tions System have been conducted such as the Base Line II, 

(1) , 1966 , Rope Eval exercise, 1971, (2) Mollohan report, 

1972, (3) and the CNO ' s Industrial Advisory Committee on 
Telecommunications (CIACT), 1972 , (4). It is not the intent 

of this paper to restate the studies which have been made but 
rather to show how these studies support the need of a com- 
munications system test procedure which will perform better 
than our present subsystem level tests . 

Many of the deficiencies uncovered during one study 
(which is classified and cannot be identified in order to keep 
this paper unclassified) , were attributable to the fact that 
an adequate method of determining and monitoring a Communica- 
tion System's Performance was not available. The inability 
to locate the cause of the failure or degradation would very 
often result in placing the blame on the other station (s) or 
on the atmosphere. An element of Black Magic was introduced 
because neither end of the trunk knew where to assign the 
blame, (itself, the other station or the atmosphere). There 
was no method available to determine the system's level of 
performance. There are standards available by which to 
measure component, equipment and subsystem performance but 
there are no standards to measure the overall performance of 
a system either shipboard or shore station. The Report said 
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there is a need to prepare and publish detailed standard 
procedures for operations , maintenance and management of 
Naval Communications on a systems level. The Report also 
pointed out that the "systems approach" is not used or taught 
at any level of the Naval training establishment. 

The studies mentioned have attempted to measure the Naval 
Commixnications System's ability to communicate and the con- 
clusions are that the system is a very poor performer. It 
takes too much time, uses too many assets, looses messages 
and, occasionally, ceases to function. 

The CIACT report called upon the Navy to develop an over- 
all plan to include a general performance description for the 
total Naval Communications system [4, p. 17]. It can easily 
be argued, that some means must be available to measure the 
"general system performance" or the "description" is of 
little value. This implies the need for a system test pro- 
cedure. The CIACT goes on to say the Naval Communications 
Command is not allowed billets for communications/electronics 
engineers to operate, service and manage a complex communica- 
tions system. With no person ready or trained to manage a 
communications system, system accountability was not called 
for, therefore, there was no test of the system in order to 
measure it. 

Now that official attention has been drawn to the need of 
a system test procedure to measure the level of performance 
of the system, an attempt will be made to specify what the 
system test procedure should accomplish. 
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The test procedure should be able to determine if the 
total system is operating and if so, at what level. No 
system will operate continuously at 100% and if System A is 
operating in a degraded mode, will it be able to communicate 
with System B which is also operating in a degraded mode? 

How much degradation can the entire system accept and still 
perform satisfactorily? The test procedure must be able to 
determine how much degradation (either in the input signal, 
or within itself) , a system can tolerate. The test procedure 
should be able to predict the system availability so that the 
user can plan the execution of this assigned mission. 

NASA has established long-term average system performance 
goals for the various transmission media (microwave , land 
line, submarine cable, communications satellite, and HF radio). 

NASA circuit availability objectives expressed on percent- 
ages, indicating the portion of the total scheduled operating 
time during which the circuit is expected to be available. 



Microwave and landlines 99.5% 

Submarine cables 99.5% 

Communications Satellite 99.8% 
High-frequency radio 95.0% 



Table 1-1 NASA circuit availability; adapted from 
Stelter [5, p. 274] 

These goals, in principle what the CIACT report calls for, 
are expressed as percentages , indicating the portion of the 
total scheduled operating time during which the circuit is ex- 
pected to be available to the user. "Circuit" as used here. 
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includes the medium of transmission and the equipment at 
both ends; a trunk. NASA measures its circuit performance 
by subjecting it to a bit error rate test of 24 hours duration 
on a periodic schedule. A Data Transmission Test Set gener- 
ates a 2047 bit pseudo random pattern that is transmitted via 
the modems normally associated with a particular circuit, to 
a compatible receiver. ■ The receiver is attached to a Print- 
Punch recorder. This recorder provides a time-tagged, perma- 
nent record of the circuit measured bit error rate. In this 
manner a circuit level of performance is established [5, p. 
272] . 

Compiling circuit performance levels enables NASA to 
establish a system level of performance. 

A Naval Underwater System Center study has proposed a 
similar test procedure. The test is made by applying an in- 
put sequence X such that the resulting output sequence will 
be Z if, and only if, the circuit is operating correctly. 

In this manner a detailed knowledge of the logic elements , 
the mechanics, the theory and their interaction used to im- 
plement the system is not necessary in order to test it. The 
test will detect almost all malfunctions resulting from 
single, simultaneous or unanticipated failures. [6, p. 4]. 

The final report on Evaluation of the Unencrypted Data 
Link System of the P-3C airplane (INSERV REPORT) brings out 
the need for a system test procedure which can isolate fault 
to a component. It criticizes the AN/ACQ-5 built-in test 
equipment which isolates faults to a group of modules or sub- 
systems which can contain up to 46 components. [7, p. 23] 
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The test procedure should be able to locate the cause of 
the degradation or failure down to the component or piece of 
equipment level in order to ensure maintainability and high 
system availability. 

In summation, the Communications System test procedure 
should be able to accomplish the following: 

1) Detect any significant malfunction. 

2) Quantify the level of performance or determine the 
level of degration at which the system is operating. 

3) Determine how much degradation in the input signal a 
system can tolerate and still perform satisfactorily. 

4) Locate the cause of the malfunction down to the 
component piece of equipment level. 

5) Aid in improving system maintainability. 

A test procedure which attempts to satisfy these criteria 
is presented in this paper. Following chapters will attempt 
to show why a test procedure is needed, what a test should 
measure and how various present test procedures compare. 
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II. A STUDY OF AVAILABILITY 



The ultimate goal of any system or equipment is to fulfill 
the particular mission for which it was designed. Before any 
system or equipment can fulfill its mission, it must be avail- 
able. Increased equipment complexity, new performance re- 
quirements, and extreme environments have resulted in higher 
failure rates, greater requirements for maintenance and lower 
availability of current systems. [8, p. 117]. There has been 
a steady improvement in component reliability, but at a slower 
rate than the increase in complexity. So systems and equip- 
ment reliability has fallen to the point where the system 
often is not available to the user when and where it is re- 
quired. Furthermore, the systems and equipment are so ex- 
pensive that it is no longer practical to get around this 
problem by buying more systems than are needed. The operators 
have come to realize that the systems they have, must be avail 
able if they are to fulfill their particular mission. [9, p. 
12 - 1 ]. 

In the 50 's and early 60 's system availability was synon- 
ymous with reliability. The prevailing attitude was that if 
enough reliability could be built into a system its correspond 
ing availability would ensure system effectiveness. This 
attitude was fostered by NASA because it determined that in 
space work there would be no means to replace or maintain any 
system which had failed. NASA utilized redundancy of critical 
components to attain the needed level of reliability. 
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Therefore reliability was equal to availability and all of 
the literature of the period stresses this point. As systems 
engineers have learned to understand and work with reliability 
they have realized that the initial premise is true but that 
systems availability could be obtained by means other than 
reliability such as maintainability. 

It must be recognized that systems availability not reli- 
ability is the ultimate criteria and that it can be obtained 
through reliability and maintainability. This criteria can 
be expressed mathematically as the relationship between 
availability A, reliability; mean time between failure (MTBF) 
and maintainability; mean time to repair (MTTR) . [10, p. 8-11]: 

A = MTBF 

MTBF + MTTR 

This paper supports that premise and will attempt to show that 
adequate testing is needed to support maintainability in order 
to achieve availability. (Fig. 2-1) 

4 

A. RELIABILITY 

Today, although in an environment of declining "technology 
spending", especially related to defense and aerospace, empha- 
sis is focused on operational and system effectiveness . Aus- 
terity in spending, coupled with the need for more sophisticated 
equipment, focuses an even more important light on the need 
for a new and more practical reliability engineering technology. 
The continued criticality of effective equipment performance in 
sophisticated systems creates a changing emphasis on reliability 
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and requires much more careful control of the design process. 
[11, p. 7-1]. 

Reliability definition: 1) The ability of an item to 

perform a required function under stated conditions for a 
stated period of time. 2) The characteristics of an item 
expressed by the probability that it will perform a required 
fvinction under stated conditions for a stated period of time 
[12, p. 198]. This general definition is just a starting 
point; the product specification for each product should in- 
clude clear definitions of all modes of operation important 
to the function of the device. Dormant and operating environ- 
ments as well as life expectancy are necessary considerations 
in the specification. 

A satisfactory definition of reliability in the product 
specification should provide all the information necessary to 
judge whether device operation is proper or improper. The 
definition should state what constitutes failure and the con- 
ditions under which failure occurs. Reliability is usually 
stated as the "mean time between failure" - MTBF along with a 
confidence interval on the mean time. 

In order to ensure th^at a system has the required reliabil- 
ity, DOD has found it necessary to establish a formal, four 
phase, system life cycle. The system consists of Concept 
Formulation, Contract Definition, Acquisition and Operational 
phases . 

In the Concept Formulation phase, the mission of the system 
is justified and its long range goals are stated. The mission 
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is defined as reliability requirements predictions are made 
on the basis of the functions that must be performed to 
accomplish system effectiveness and mission successes. The 
accuracy of these predicted requirements need only be enough 
to justify proceeding with the next phase of the project. 
Alternative solutions are listed and feasibility studies may 
be conducted. 

The Contract Definition phase includes identification 
and specification of system characteristics and the prepa- 
ration of program plans which will assure a high probability 
of achieving the specified missions in a cost effective 
manner. Modern program management demands that specification 
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_J , 

MAINTAINABILITY 



(Operating time) (down time) 

! 

TEST PROCEDURE 



AVAILABILITY 
- Adapted from Blanchard [8, p. 18] 

(Fig. 2-1) 



decisions be based on quantitative analysis. Numerical 
values for the system characteristics, i.e., relicibility , 
maintainability, operability, safety, cost analysis, quality, 
security, training, procedures and logistics, must be predicted 
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during the Definition phase of the system life cycle. Reli- 
ability and maintainability program plans must be part of a 
total management system. Identification and funding of reli- 
ability and maintainability programs must be an integral part 
of a program managers overall cost, schedule and technical 
control system. [13, p. 1-5]. These characteristics must be 
repredicted or assessed repeatedly throughout the Acquisition 
phase and they must be measured or demonstrated before and 
during the Operational phase. These numerical values have 
value only when used to give a predicted, assessed or mea- 
sured value of overall system effectiveness. 

While determining a quantitative reliability specification, 
certain factors must be kept in mind. It is impossible for 
man to build a system with 100% reliability because of the 
limits of knowledge, physical laws and monetary constraints. 
There will be a reliability decrement due to design defi- 
ciencies, manufacturing deficiencies and use. (Fig. 2-2) 

These losses will occur. In addition to the physical environ- 
ment problem the effect of the social environment containing 
people,' the man/machine interface, and their expectations, 
goals, motivations, training, and other psychological and 
physiological attributes will introduce some variance to the 
stated reliability. [15, p. 255]. The system planners must 
determine how much reliability loss can be expected and how 
much can be tolerated. There are numerous mathematical 
methods of quantifying the reliability necessary for a par- 
ticular system. Such as Monte Carlo models [16, p. 4) and 
regression analysis. 
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An additional input needed before a level of reliability 
sought can be selected is cost. It must be recognized that 
there is a cost effective level of reliability and that any 
demand for greater reliability will be disproportionate with 
' its cost. Above this point some means to achieve increased 
availability other than reliability should be sought. Trade- 
offs between cost, availability, relial)ility and maintain- 
ability must be analyzed to determine the cost effective 
blend. 

With quantitatively designed reliability values available 
manufacturing can be monitored to ensure attainment of the 
needed reliability levels. Emphasis must be placed on the 
fact that the reliability goals are determined prior to 
manufacturing and are included in the specifications. In 
this manner all personnel working on, and the ultimate users 
of the item are aware of the established reliability levels 
and can work to assure their attainment. This assumes the 
existence of 1) testing or verification procedures to deter- 
mine the actual level of reliability and 2) management control 
procedures to handle any variance between the desired and 
actual levels of reliability attained. The management con- 
trol procedures should be specified in a management plan. 

While reliability considerations can be very useful they are 
not a panacea as the following examples will demonstrate: 

1) When using MTBF as a reliability criteria how is a 

component which does not fail but goes into degraded 
operation handled. It is not a black and white but 
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a grey area. If specifications were written to cover 
all grey areas they would be voluminous . 

2) Determining reliability and cost estimates for 
marginal analysis is expensive and time consuming. 

3) At present the performance of some systems is difficult 
to quantify and does not lend itself to quantitative 
values which can be utilized for reliability specifi- 
cations; i.e., voice communications systems. The 
object of the system is to pass intelligible infor- 
mation. How is "intelligible" defined? MIL-STD-188C 
revised 24 Nov. 1969 gives the following performance 
objectives. "Tactical transmission system volume, 
loop losses , switch losses and inter-office trunks 
shall be engineered to provide the following listen- 
ers preference ratings in terms of listeners prefer- 
ences: 

85% of the users will rate the circuit "Good". 

10% of the users will rate the circuit "Fair". 

5% of the users will rate the circuit "Poor or 
Worse"." [17, p. 70] 

Although these are performance criteria, they do not satisfy 
the criteria of quantitative, reliability specifications. 

B. MAINTAINABILITY 

Maintainability definition: A time function, the prob- 

ability, understated conditions, that a maintenance process 
will terminate successfully within a stated time measured 
from initiation of the process. 
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The maintenance need can be brought on by degraded 
operation, catastrophic failure or because of the lapse 
of a predicted reliability time period. The maintenance 
process can be preventative maintenance which considers time 
as a constant, either because regular periods of fixed 
duration are provided for the purpose or because the tasks 
are so routine that performance times exhibit negligible 
dispersion. On the other hand corrective maintenance typi- 
cally involves unforseen contingencies such as catastrophic 
failure or systems deterioration while still performing above 
a "failed" level [18, p. 137]. 

Reliability is one side of the system "availability" 
coin. Though complex equipment can fail, reliability has 
attained a degree of sophistication that makes it possible to 
predict failure rates with a fair degree of accuracy. The 
other side of the coin, maintainability, presents a problem. 
Even though the need for maintainability has been recognized 
in some quarters it has not received wide attention and 
therefore has not reached the stage where it can be effec- 
tively quantified with any appreciable degree of acceptance. 
Since most things are fallible, maintainability, or the need 
for it, is a fact of life. When failure occurs repair is 
required and the faster the better. If either "ability" 
fails to do its job, or fails to support the other, avail- 
ability suffers. [19, p. 1-5]. 

A useful description of availability is contained in 
NAVSHIPS 94324. The mathematical approached used is 
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A = R + Mq (1-R) 

where A is the availability, R is the equipment reliability 
and 1^ is the operational maintainability of the equipment. 

[20, p. 3]. 

The initial Concept Formulation phase, when a future 
system is being considered and the alternatives are being 
examined, is when maintainability long range goals should 
first be considered. From the point in time when reliability 
becomes a consideration maintainability must also be present. 

In the Concept Formulation phase, maintainability functions 
such as, logistics, support systems, personnel and training 
must be present and kept visible throughout the system 
evolution . 

Maintainability requires an integration of design and 
maintenance engineering effort to provide controls which 
assure that the system being designed is adequately, expedi- 
tiously, and economically supported. Maintainability is a 
design parameter which reflects support considerations, and 
is prerequisite to integrated logistics support. [8, p. 117]. 

Quantitative maintainability values must be determined by 
considering trade-off's with reliability and cost. Figure 
2-3a represents trade-off alternatives between reliability 
and maintainability at a constant availability level. 

Figure 2 - 3b shows the effect upon reliability and main- 
tainability of cost and Figure 2-3c depicts the trade-offs 
between availability, reliability, maintainability and cost. 
During this phase the system characteristics must be determined 
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COST-AVAILABILITY TRADE-OFF CUR^/ES 




Roliability 
Figure 2-3A 

Equal Availability Curves 




Figure 2-3B 
Equal Cost Curves - 




Reliability 
Figure 2-3C 

Optiimim Cost-Availability Curve 

Adapted from Eiostad (21, p, 337) 
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and used as guidance. The basic methods of construction 
(extent of modularity, accessibility, fastenings, plug-in 
and wire-in features, etc.) , the location and circuit protec- 
tion of test points, the level of the lowest replaceable unit 
(LRU's), the extent to which automatic fault detection (AFD) 
and built-in test equipment (BITE) will be provided, the type 
and extent of fault isolation and localization software 
(programs for AFD and technical manuals and data for main- 
tenance men) , and the nature and degree to which on-line 
performance monitoring and testing is to be provided, must all 
be specified during the Contract Definition phase. [21, p. 33] 

Various models have been developed based upon mathematical 
distribution models to aid in quantifying maintainability 
[22, p. 501] [18, p. 137] [20, p. 3]. 

« 

If we are to profit from the ground work performed during 
the previous phases , the Acquisition and Operational phases 
must.be guided by two paramount considerations. First we 
must be prepared to perform the management function of con- 
trol. Controlling is the management function of making sure 
that plans succeed. In other words, it is the measuring and 
correcting of activities to ensure that these activities are 
contributing to the achievement of planned goals. [23, p. 507] 
During the Concept Formulation and Contract Definition phases 
standards for maintainability were established; during Acqui- 
sition and Operations maintainability performance must be 
measured and deviations from the standards corrected. 
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Measurement is the second paramount consideration. Since 
we are concerned with systems availability which we plan to 
enhance through systems maintainability, we must have a means 
to test the systems maintainability to be able to quantify 
the level of performance. 

Test procedures which test the entire system and not 
components of it, are very hard to find, and are the subject 
of this paper. 



23 



III. MODES OF FAILURE 



This chapter will attempt to analyze failure; its causes, 
types, occurrence and effects. 

A. CLASSIFICATIONS OF FAILURE 

Failures can be classified according to location or cause. 

A classification of defects according to the location of 
failure determines a weak point in a system and is the first 
step needed to strengthen it. The reason for the occurrence 
of the failure can be analyzed as follows : 

Design defects: Defects attributable to design imper- 

fections. Pushing the state of the art or failure to consider 
all possible stimuli and responses of a system lead to design 
defects . 

Manufacturing defects: Failures of this class occur as 

a consequence of violation of the technological manufacturing 
procedure chosen for the system or unit. The quality of the 
individual units and components of a system have unavoidable 
random variations. These manufacturing variations, decrease 
the reliability of some of the items or units. 

Improper use: For every system, restrictions are made 

on the conditions of its use. Rules are given for maintenance 
and operation of the system and its parts. Violations of these 
rules or restrictions may lead to premature failure. 

Deterioration (aging) : No matter how high the quality of 

the unit and/or the system as a whole, gradual wear or aging 
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is inevitable. During the course of use and storage, ir- 
reversible changes take place in metals, plastics and other 
basic materials. The cumulative effect of these changes 
deteriorates the strength, coordination and interaction of 
the parts. This deterioration may not be as readily apparent 
as an outright failure. The piece of equipment or system is 
still operating but at a sub-optimal level of performance. 

All components, subsystems and systems are siibject to dete- 
rioration which in the final analysis may cause failure. 

B. TYPES OF FAILURE 

Instantaneous failure: Failures occur at the instant an 

operational parameter is exceeded. These failures are random. 
Typically the failure of the unit or system occurs independent- 
ly of how long it has been used or what condition it is in. 

An example of this , is connecting a 110 volt appliance to a 
440 volt power supply. The failure is instantaneous and 
usually readily apparent, no test may be necessary. 

Degradation failure: This failure occurs as the result 

of the gradual change in operating parameters. Admissible 
limits for the operating parameters of a unit or system are 
established when it is designed. A performance parameter 
operating outside these limits is classified as a failure. 

[24, p. 5]. All units or systems are subject to wear or aging 
and therefore their condition is continually changing or 
transient; usually it is deteriorating. A minimum performance 
limit is established and the unit or system will approach this 
limit throughout its lifetime. This state is known as 
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"graceful degradation." A test procedure is needed to 
determine how much degradation has occurred. 

Interaction failure; A situation in which several causes 
of failure act simultaneously. It is the most typical type 
of failure in practice. A vacuum tube may fail because it 
was subjected to a random "peak" load, after graceful de- 
gradation had lowered its upper level limit. The result is 
instantaneous failure. Interaction failure can be reduced by 
calibration which restores the upper limit to its proper 
level, if the amount of degradation is known. 

C. OCCURRENCE OF FAILURE 

The occurrence of failure is examined by analyzing the 
distribution of failures with respect to time, especially for 
components utilizing electronic technology. Figure 3-1 shows 
the classical failure rate for a finite population of newly 
manufactured components under a uniform operating stress. 

The failure rate curve is referred to as the bathtub curve. 

It is based on the presumption that failures will be fixed 
as fast as they occur, to keep the component operating. 
Virtually all components start their life with a very high 
initial failure rate, largely because of design and manufac- 
turing deficiencies which economical quality control cannot 
find. This period is known as a debugging, or burn-in or 
infant mortality period. Following the burn-in period, the 
component enters its useful life period. Toward the end of 
the useful life, so many parts are deteriorating so fast that 
maintenance becomes uneconomical, the failure rate begins to 
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rise and the component enters its v;ear-out phase or end 
of life [25 , p . 2 ] . 

The classical failure rate has been challenged with respect 
to its validity from the user's point of view as applied to 
electrical components. Most contracts call for a manufactur- 
er's burn-in period prior to acceptance. For the user the 
burn-in period is nearly non-existent. With integrated 
circuits, printed circuits, chips, transistors, etc., most 
electrical components are obsolete and are replaced before 
they enter the wear-out phase so it does not apply. 

Examination of the useful life period calls for following 
a subtle line of reasoning and the solving of a contradiction, 
which this paper will not attempt. The first arg\iraent says 
that, given adequate preventative maintenance, experience 
has shown that the failure rate is fairly constant during 
this period. The second argument says that since the failure 
rate during the useful life period is determined by random 
failures preventative maintenance will have no effect upon 
the failure rate and therefore serves no purpose. Both 
arguments accept useful life failure and the existence of 
degradation which may impede a component's performance while 
not lowering it to a "failure" status. 

F. EFFECTS OF FAILURE 

The effects of failure can be as varied as the user, the 
component or the system which has failed. The effect could 
range from insignificance to catastrophy and loss of life. 
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In order to avoid the effects of failure, missions are 
curtailed or altered, duplicate systems (redundancy) are 
employed and several more expensive solutions are implemented. 
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IV. PRESENT TEST PROCEDURES 



Before examining test procedures, equipment levels will 
be defined. 

Component: A component, (subassembly, assembly, unit or 

group) is any collection of pieces which composes a sub- 
division of a set, piece of equipment or system. By itself 
it is capable of independent operation but it is not capable 
of performing a complete operational function. It is usually 
replaceable as a whole, but has parts which may be individu- 
ally replaceable. (Examples: Electronic power supply, radio 

receiver, antenna group, oscillator). 

Equipment" (set) : A component or components and necessary 

parts connected or associated together to perform an opera- 
tional function. (Examples: Radio receiving set, sound 

measuring set, which includes such parts and components as 
cable, microphone and measuring instruments; radar homing 
set) . 

Subsystem: A subsystem is defined as a combination of 

equipments, components, etc., which perform an operational 
function within a system. 

Subsystems from the major subdivisions of systems. 
Examples: (a radar station, fire control subsystem, a radio 

transmitting facility. 

System (Electrical-Electronics) : A combination of equip- 

ment and/or subassemblies , generally physically separated 
when in operation and other such components necessary to 
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perform an operational function or functions. Examples: 
(Antiaircraft defense system including tracking radar, computer 
and g\m mount; GCA electronic system; communications system; 

A link including transmitter, medium of transmission and 
receiver). [26, p. 4]. 

In order to determine the level of operation of a test 
procedure we must specify the level of operation of the item 
being tested. The composition and the characteristics which 
must be measured at each level must be specified. Once the 
characteristics being tested have been defined, the level of 
operation of the item and the test system can be defined. 

Table (4-1) attempts to categorize and arrange the equipment 
levels and the characteristics measured. 

A capsule simmary of the prominent test procedures used 
in Naval Communications will follow. It must be understood 
that variations in tests occur from ship to ship and fleet to 
fleet, however an attempt will be made to provide a basic, 
objective guideline and discussion of each procedure. 

The Quality Monitoring System (QMS) is a subsystem level 
performance monitoring procedure. It is composed of hard 
wired, monitoring equipment which is interfaced into the 
commimi cations system at key points. It can be used on-line 
or off-line to measure degradation of selected character- 
istics. It requires extensive training and is expensive to 
install, however, quantum improvements in communications 
system performance can be obtained when QMS is properly used. 
QMS is capable of performing qualitative, voice, systems 
level testing. [27] 
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EQUIPMENT MEASUREMENT LEVELS from Thomas & Clark 



Pomsee is a subsystem level calibration test. It can be 
performed aboard ship but its best application is ashore at 
a depot level maintenance facility. Equipment must be removed 
and dismantled and is tested on a test bench. It is costly, 
time consuming and requires a high level of training to be 
properly performed. It is best applied to pieces of equipment 
which are known to have failed and not as preventative main- 
tenance or in order to locate a failure. 

Built-in test equipment (BITE) is a confusing title to 
apply to a test procedure. QMS could be considered built-in 
test equipment, however the term usually applies to component 
or equipment level "self-test" procedures. BITE is capable 
of being applied at the subsystem or system level, however, 
it is not done in practice because of cost. It usually pro- 
vides a GO /NO-GO indication, a continuity check of cables 
and connections and requires an insignificant amount of time 
to perform. No special training is required to use the test. 
Its primary draw-backs are cost, one test circuit or piece 
of test equipment is required for each item being tested, 
and the sparcity of trouble shooting information available, 
it usually won't say what component failed or why. [28, p. 

4-1] 

The Standard Measurement Technique (SMT) is a quantitative, 
system level, test which measures the flow of information 
being processed by the system. It can be hard wired or 
patched in. It is an inplace, off-line test which is capcible 
of qucintifying the system cibility to process degraded inputs. 
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It requires very little training, is moderately expensive 
and can be performed in minutes. SMT is being developed by 
the Naval Electronic System Command and a complete description 
of the system is provided in the following chapter. 

The Planned Maintenance System IPMS) is a s-ubsystem level, 
off-line GO /NO-GO test. It can be used on a daily basis and 
requires B-school training to perform. Portions of it can be 
performed with less training and portions will require higher 
training, more time and a test bench. PMS can be hard wired 
into the system but it is usually patched in, using portable 
equipment. The use of portable test equipment presents a 
problem because the equipment is often "banged" out of cali- 
bration rendering it unreliable. PMS is used to locate a 
failed piece of equipment. The 3M cards are used with PMS. 

The 3M test procedure is a subsystem level maintenance pro- 
cedure. Basically it is a check list, set of instructions 
listed on a card which describes a test, lists the equipment 
necessary and gives step by step instructions on how to perform 
the test. It can be performed off-line, or on a test bench. 
Fleet studies have indicated that component and equipment level 
3M/PMS testing is performed in the fleet but that subsystem 
level testing is not performed. [29] 

Table 4-2 presents the various test procedures and attempts 
to show the significant criteria which can be used to compare 
them with each other. 

The definition of each criteria used in Table 4-2 is as 
follows : 
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Component, equipment, subsystem and system - as presented 
at the beginning of this section or Table 4-1. 

Detect failure - can the test procedure detect a failure 
at the highest level of performance given? 

Detect degradation - can the test procedure detect de- 
gradation at the highest level of performance given? 

Quantitative - does the test procedure provide a quanti- 
tative read-out (as opposed to a GO /NO-GO type output) at the 
highest level of performance given? 

Measure flow of information - does the test procedure 
have the ability to measure the flow of information in the 
system? 

Performed daily - is the time needed to perform the test 
such that it can be performed daily? 

Remove and replace - does the item to be tested have to 
be removed in order to be tested? 

High training - is a high level of training (B-school) 
necessary in order to be able to perform the test? 

It must be pointed out that table (4-2) does not consider 
the cost of each test procedure. Cost analysis must be con- 
sidered before any test can be considered optimal. 

The items which should be given major consideration are 
System, measure flow of information, performed daily and high 
level of training. These are the items which the user of the 
test equipment is interested in and upon which his judgment 
will decide if he will use the test procedure. 
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TEST PROCEDURE COMPARISON 



V. STANDARD MEASUREMENT TECHNIQUE 



A. GENERAL CONCEPT OF STANDARD MEASUREMENT TECHNIQUE 

Studies and communication exercises performed over the 
years have pointed out again and again, the fact that an 
adequate method of determining communications SYSTEM perfor- 
mance does not exist. This chapter will attempt to offer a 
solution to this problem. 

Performance evaluation has been confined to a lengthy 
and complicated (POMSEE etc.) evaluation of individual equip- 
ments and proceeded on an assumption that proper operation of 
system components ensured satisfactory operation of the total 
system. This viev/point ignores the fact that systems are 
often synergistic and presupposes component performance levels 
are linear and arithmetically additive. 

The emphasis on individual equipment performance often 
resulted in catastrophic equipment failure as the sole criteria 
for maintenance action while quantum system performance de- 
gradations went largely unnoticed. [30, p. 3 - 1 to 3 - 17] . 

A major weakness of the conventional methods for testing 
and maintenance of communications systems is the inability to 
test for varying levels of degradation in the system. Attempts 
to establish a communications circuit using ostensibly opera- 
tional equipments often resulted in failure. Diagnosis of 
circuit malfunctions did not isolate faults to the ship or 
shore termination, the atmosphere, or the identification of 
faulty system components at the specified site. 
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The genesis of some of these problems can be found early 
in the equipment life cycle. During acquisition equipments 
are tested by various parameters , levels , procedures and 
conditions. The ambiguities in equipment performance levels 
often limit the amount of predictive analysis a communications 
systems engineer can perform prior to system implementation 
resulting in performance requirements written post facto. 

Subsequent system testing (OPCAP, INSURV, etc.) frequently 
utilize a shore termination as an operational standard. Since 
the standard has not been tied to a reference point utilizing 
strict methodology the predictive performance aspects of the 
system operating with different terminations often suffer. 

A need for formalized testing methodology, from equipment 
level to systems level, is evident. An additional necessity 
is the establishment of minimum system performance levels. 

All too often system levels are established at or near design 
levels (as new or refurbished equipments) without an adequate 
determination of allowable performance deterioration due to 
aging. As a consequence, equipment replacement or overhaul 
can become a qualitative, judgment without adequate consider- 
ation of an equipments' effect on its suit or ancillary 
equipment . 

The Standard Measurement Techniques (SMT) will fill 
this long-standing need. The SMT will provide ships and shore 
stations with techniques and procedures for testing and 
measuring overall communications effectiveness on a system, 
subsystem, equipment, and circuit level without an emphasis 
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on automation, the need for general purpose digital com- 
puters, or highly trained operators and technicians. With 
the use of magnetic tape inputs and the traditional approach 
to communication testing, a valuable new technique is developed 

The evaluation of system operation can be determined rap- 
idly by the following technique: Prerecorded teletype signals 

with various levels of degradation, such as distortion and 
atmospheric disturbance will be available on magnetic tapes. 

The tapes allow a communication system to be checked under 
varying conditions that are experienced' in actual ship and 
shore operations . 

The National Bureau of Standards and the Pacific Missile 
Range will pretest, verify and supply the magnetic tapes. 

The various test signals and the utilization of existing 
displays will permit interpretation on, a GO/NO-GO basis, the 
ability of the system to meet functional specifications. 

To provide for universal testing it is necessary to 
accomplish the following long range goals: 

a. Define and describe present systems. 

An Engineering Data Register (EDR) has been promul- 
gated to define the pertinent characteristics and 
specified parameter values for each Naval Communica- 
tions System. 

b. Establish numerical systems performance levels. 
Reference Level Measurement Guides (RLMG) will 
establish acceptance levels criteria for systems 
and equipment. They will specify the following 
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for a particular communications system: 

- which parameter is to be tested. 

- the value or level for measured parameter. 

- which measurement method in the SMT should be used, 
c. Develop techniques for measuring system performance 

levels. The SMT defines the applicable technique 
for measuring each characteristic in conjianction 
with testing the overall system performance. 

In a letter to the Chief of Naval Operation (CNO) , dated 
21 Sept. 1972, the Commander, Naval Electronic Systems Command 
stated the following: 

"The purpose of SMT is two-fold. First, it measures 
communications system's performance to determine system's 
operational capability. Secondly, it has the capability to 
locate or identify equipments that are not performing properly. 
The tests are conducted on record, and voice communications 
systems londer simulated, controlled, repeatable conditions 
that exercise all parameters of a system. The test can be 
performed at sea or in port without need for assistance from 
other ships or shore stations." [31]. 

In order to contrast the SMT approach with present testing 
methods, the following distinctions are made: 

Philosophy of present testing methods: The basic principle 

of the current testing methods is that the better a system is , 
the less it will distort the most perfect signal. Tests are 
conducted for distortion of a single characteristic. 
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Philosophy of SMT testing methods: The basic principle 

of testing methods with SMT is that the better a system is , 
the greater degradation of input it will withstand and con- 
tinue to operate. Tests are conducted for system response 
to maximum distortions of multiple characteristics. [32] 

Six types of system degradation are examined with SMT; 

. Teletype Distortion; Teletype systems are tested by 
introducing five lines of test in successively increased 
amounts of distortion from a tape, each line containing a 
different type of distortion. The terminal teletype of the 
subsystem is then checked to determine at what point the 
system fails to produce an error-free copy. 

. Atmospheric Distortion: Teletype systems are tested 

by introducing ninety lines of test with Multipath, Noise and 
Fade from a tape to check the system's diversity capability 
and the receiver subsystem's response to signal amplitude 
fluctuation and sensitivity (S/N) . The terminal teletype of 
the system/subsystem under test is examined to determine the 
system's capability to produce an acceptable copy. 

. Transmitter Spectral Purity; A system's transmitter 
is tested by examining its output with a spectrum analyzer 
for percentage of modulation, harmonic distortion, inter- 
modulation distortion, and suppressed carrier level. 

. Articulation Index (AI) : Voice systems are tested by 

inputing various audio frequencies and a precise amount of 
noise from a tape. The receiver output is then measured 
with a noise measuring set to determine the intelligibility 
capability of the system. 
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. Multi coupler/coupler Swept Frequency Response: A 

system's multicoupler is tested by introducing various 
frequencies from a tracking generator into the system's multi- 
coupler and examining its output with a spectrum analyzer to 
determine loss or gain, bandpass, ripple factor and skirt 
characteristics . 

. Antenna Swept Frequency Response: A system's antenna 

is tested by introducing a continuous spectrum sweep from a 
tracking generator at the antenna through a directional 
bridge. The output is then examined on a spectrum analyzer 
to determine the standing wave ratio and resonant frequency 
of the antenna. The antenna signature presented on the 
spectrum analyzer is photographed and maintained as a main- 
tenance standard. [32] 

B. SPECIFIC OBJECTIVES OF SMT 

We are primarily concerned with the questions of whether 
communication equipment is operational, and at what degradation 
level it becomes inoperable. Therefore, the scope of the 
Standard Measurement Techniques program involves four levels 
of testing, each progressively more detailed. Tests are: 

1. SYSTEM TEST 

2. SUB-SYSTEM TEST 

3. EQUIPMENT TEST 

4. CIRCUIT TEST 

The tests are designed with the following objectives: 

a. Tost for system response to maximum distortion of a 
multiple of characteristics with a minimum of tests. 
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Reliability; tests are to duplicate real world 
coitimuni cations environments and will measure system 
performance under worst-case conditions. Systems 
orientation; emphasis on providing realistic simulated 
inputs to systems and observation of resultant outputs. 
Define numerical system parameters for inclusion in a 
Re f e ren ce _ Le ye 1 Me asureme njt Gui de (RLMG ) and establish 
acceptance/rejection criteria. Performance level 
indices; numerical analysis of system performance vice 
a reliance on GO /NO GO judgments to provide a measure- 
ment of system quality. 

b. Utilize the existing system equipments to interpret 
results. Economy; an avoidance of automation, com- 
puters, and highly trained personnel. Design goal is 
to provide total SMT capability for less than $5000.00 
per site. 

c. Minimize the number, sophistication and time of tests 
and technicians. Speed; complete systems tests in 
less than 15 minutes per test. Simplicity; test 
procedures that are straightforward and easily under- 
stood by operational personnel at the fleet level. 

d. Maximize confidence in the validity of system acceptance. 

e. Establish capability to rapidly and confidently as- 
certain particular system, subsystem or equipment 
characteristics which do meet specified parameter 
values or tolerances . 
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f. Commonality for all communications systems. Inde- 
pendence; performance of SMT measurements that are RF 
silent, self contained, and can be accomplished with- 
out support of off-site activities . Devise and 
establish Standard Measurement Techniques (SMT) for 
testing and measuring system performance levels of 
communications systems employed at communications 
stations afloat and ashore. Develop and prepare 
Detailed Test Plans (DTP) tailored to a specific 
platform/site and establish procedures for reporting 
results of testing performed. [33, p. 2] 

C. TEST PROCEDURES 

The RLMG describes system tests necessary to ascertain 
that equipment characteristics meet the specifications listed 
in the EDR. 

Test Notes 

'a. Each test procedure is furnished with a block diagram 
for use during test set-up. The test equipm.ent is 
shaded for clarity. 

b. A one hour warm-up period is recommended to stabilize 
the system and test equipment. 

c. Safety Procedures; NAVSHIPS 250-660-42, Electric 
Shock Its Causes and Prevention, should be made 
available to personnel engaged in testing. 

d. Equipment that is essentially passive, e.g., receive 
and transmit switchboards and interconnecting lines , 
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is assumed to have been tested prior to start of 
system tests. [34, p. 1-2]. 



Functional System Tests (Figure 5-1) can be used for 
systems orientation . 

System Test Part 1. TTY end equipment. 

System Test Part 2. System Test with TTY Distortion 
System Test Part 3. Receive System Test with Atmospheric 

Distortion 

System Test Part 4. Antenna (Transmit and Receive) 



System Test Part 1: TTY end equipment 

System Test No. 1 is a teletype "end equipment" functional 
check of reperforator and page printer only on a closed loop 
basis via the DC patch panels. Teletype distortion tape: 

Five lines of test, each line containing a different type of 
distortion, line 1, MARK BIAS, line 2 SPACE BIAS, line 3 
SWITCHED BIAS, line 4 SPACE END, and line 5 MARK END will be 
transmitted in successively increased amounts of distortion, 

5% increments, through 45% distortion. The end equipment or 
teletype of the system/sub-system under test will be checked 
to determine at what point the teletype machine fails to 
produce an error-free copy. [35, p. ii] 

Procedure details for system test part 1: 

a. Patch the tape recorder to the DC patch panel. Patch 
DC patch panel to page printer. 

b. Send type test message; TEST THE QUICK BROWN FOX JUMPS 
OVER THE LAZY DOG'S BACK 123456789jl testing. 

c. Verify printed test message at page printer. 
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Flguro 5-1 Functioned System Block Diagram 
^^ultichannol, Duplex, AFTS, RATT 
•'P” System 
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d. Repeat test message for each channel of "P" system. 

e. Objective of the test is to ensure proper operation 
of perforators and page printers. Figure 5-2 is a block 
diagram of the test. Table 5 - 1 is an example of the 
printed page messages. 

System Test Part 2: System test with TTY distortion 

System Test Part 2 is a quantitative semi-full system 
loop test. The following technique will demonstrate the over- 
all system performance and its capability to perform with 
known distorted signals. Input to the system is provided by 
a magnetic tape recorder. The tape recorder will have four 
(4) instrumentation channels. Three instrumentations channels 
will have prerecorded teletype signals which are degraded at 
successive intervals with various levels and types of dis- 
tortion such as space bias distortion and spacing end 
distortion. The fourth instrumentation challel will be used 
for the tape recorder speed servo control loop. The test tape 
used in test Part 1 is used for this test. [33, p. 7]. 
Procedure details for system test Part 2. 

a. The tape recorder is patched via an interface unit, 
to the DC patch panel. The transmit antenna lead is removed 
from the antenna and patched to a dummy load. The transmitter 
is patched to the monitor equipments . The receive antenna 
lead is removed from the antenna and patched to the monitor 
equipment. This series of connections makes a loop permitting 
a test signal to be introduced by the tape via the DC patch 
panel. The test signal is sent through the transmit system 




Figxu -0 5 - 2 TTY Tost 



System Tost Part 1 "P” Syaton 
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STANDARD MEASUREMENT SYSTEM TEST TAPE 



to the monitor equipment, to the receive system to the 
page printer. 

b. Send tape test message TEST THE QUICK BROWN FOX JUMPS 
OVER THE LAZY DOG'S BACK 12 3456 789 j2f TESTING. 

c. Verify printed test message at the page printer. 

When a NO-GO condition appears maintenance work can be 
accomplished by use of the Sub-System test 1, after all 
systems tests are complete. 

d. Repeat message for each channel of "P" system or by 
using an interface panel test all channels at one time. 

e. The objective of this test is to ensure proper 
operation, compatibility and performance of the system, ex- 
cluding the two antenna systems, the effects of the propa- 
gation medium and the transmit TTY equipment. A block dia- 
gram, Fig. 5-3, is supplied for guidance, and table 5-2 
lists the equipment necessary to perform the test. 

System Test Part 3: Receive System test with atmospheric 

distortion. 

System Test Part 3 is a quantitative, receiving system 
test. The following technique will directly imply the level 
of performance to which the system will operate. Input to 
the system is provided by the magnetic tape recorder as used 
in Test No. 1. Three instrumentation channels v;ill have pre- 
recorded multichannel RATT messages which are degraded at 
successive intervals with various levels and types of atmos- 
pheric conditions such as time spread distortion and noise. 
The fourth instrumentation channel will be for the tape 
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LIST OF TEST EQUIPMENT FOR SYSTEM TEST PART 2 

[34, p. 2 - 8] 



Description 

1. Magnetic Tape Deck 

2. Interface Unit 

3. Spectrum Analyzer 

4. Frequency Counter/Generator 

5. Attenuator 120 db 

6 . Attenuator 12 db 

7. Micro Voltmeter 

8 . Dummy Load 

9. Power Dividers (2) 

10 . Line Fuse for Spectrum Analyzer 

11. Test Tapes 

12. Patch Cords 



Table (5 - 2) 
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Fig\iro 5-3 Semi-Full Loop Test 
System Test Part 2~”P" System 
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MATRIX 



recorder speed servo control loop. Atmospheric Distortion 
tape: Ninety lines of test, including thirty lines of MULTI- 

PATH distortion to determine the system diversity capability. 
Thirty lines of FADE to exercise the receive systems capability 
to respond to signal amplitude fluctuations (AGC) . Thirty 
lines of NOISE, to test the receive systems sensitivity (S/N) . 
In all cases, the end equipment, teletype, of the system/sub- 
system under test v/ill be checked to determine the systemis 
capability to produce an acceptable copy. (1 error in ten 
lines acceptable, for the system under test.) [33, p. 11] 
Procedure details for system test part 3: 

a. A synthesizer is utilized to supply an RF input to a 
mixer in conjunction with the input from the tape recorder. 

The test signal is processed through the receive system (in- 
put is below the antenna) to the page printer. 

b. Send recorded RF signals, with atmospheric distortion. 

c. Verify printed test message at the page printer. 

When a NO-GO condition appears maintenance work can be 
accomplished by use of the Sub-System 2, after all systems 
tests are completed. 

d. Repeat test signal for each channel of "P" system or 
by using an interface panel test all channels at the same time. 

e. The objective of this test is to demonstrate the 
opercibility and performance of the receiving system when 
exposed to the varying effects introduced by a test signal 
with known quantities of atmospheric distortion. A block 
diagram, fig. 5-4 , is supplied for guidance and table 5-3 is 
a list of equipment needed to perform the test. 
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Figure 5-4 Receiving System 
System Test Part 3 ”P" System 
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LIST OF TEST EQUIPMENT FOR SYSTEMS TEST PART 3 

134, p. 2 - 15] 



Description 

1. Magnetic Tape Deck 

2. Test Tapes 

3. Synthesizer 

4. Mixer 

5. Attenuator 120 db 

6. Attenuator 12 db 

7. Micro Voltmeter 

8. Power Divider 

9 . Line Fuse 

10. Patch cards 



Table (5 - 3) 
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Systems test Part 4 : Transmit and Receive Antennas 

System Test Part 4 is a transmit and receive antenna 
test. Input to the system is provided by the magnetic tape 
recorder. An instrumentation channel will have a prerecorded 
two-tone signal. This two-tone signal is supplied to the 
transmitter with a full power setting utilized for the trans- 
mit antenna. The transmit and receive antenna signals are 
evaluated at the Current Monitoring Panel (CMP) . The receive 
antenna signal is provided by the transmit loop of the "P" 
system at a minimum level to prevent multicoupler overloading. 
[33, p. 15] 

Procedure details for system test Part 4 . 

a. Patch tape recorder to the transmitter Switchboard. 
Attach a current coil to transmitter antenna lead 

and to the receive antenna lead. Connect both current 
coils to a current monitor panel. Patch a spectrum 
analyzer to the current monitor panel. 

b. Record spectrum analyzer display and determine the 
standing wave ratio and resonant frequency of the 

•• antenna. 

c. Send taped two-toned signal. 

d. Repeat test for each antenna. 

e. The objective of this test is to determine the 
efficient and inefficient frequencies (or frequency 
ranges) for each antenna. Reference Fig. (5-5) for 
a block diagram and table (5-4) for a list of equip- 
ment necessary to perform the test. 
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LIST OF TEST EQUIPMENT FOR SYSTEMS TEST 4 
[34, p. 2 - 21] 



Description 

1. Magnetic tape deck 

2. Test tapes 

3. Spectrum Analyzer 

4. Antenna current monitoring panel 

5. Current coils 

6 . Patch cords 



Tcible (5 - 4) 
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Sub-System Tests (Second level of the SMT) 

The Sub-System Tests are designed to isolate a particular 
part of the communication system when a NO-GO condition 
appears during System, Tests No. 2 or No. 3. There are two 
tests, two parts to each test. By a process of elimination, 
each test will pinpoint the NO-GO equipment or equipments 
for that particular test. 

The other components in the circuits are essentially 
passive, i.e., receiver and transmit switchboards and inter- 
connecting lines. Existing standard tests can be utilized 
for these items. The foregoing technique isolates the 
transmit and/or receive section of the overall system to en- 
able the test personnel to pinpoint the exact NO-GO equipment. 



[33, p. 18] 

Procedures for Sub-System Test No. 1 

a. Set up test and monitor equipment as in Systems test 
Part 2 except that the tape recorder is patched to 
the transmitter switchboard instead of the D. C. 
patch panel.' The tape recorder sends the test 
'•signals through the transmit section of the "P" 
system. After the transmitter is tested and verified 
the test tape input reverts back to the multichannel 
. keys and the test is run again. Transmitter Spectral 
Purity: The transmitter under test will be tone 

modulated, 8 or 16 tones, as required and the output 
will be ex£imined on a spectrum analyzer to determine 
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the spectral purity, (% of modulation, harmonic 
distortion, intermodulation distortion, and suppressed 
carrier level) that applies to the transmitter under 
test . 

b. A black diagram. Fig. (5-6) is provided and table 
(5-5) is a list of equipment necessary to perform 
the test. 

Procedures for Si±>-System test 2: 

a. Patch the tape recorder to the receiver switchboard 
and test the multichannel convertor. After the 
convertor is tested test the receiver as in System 
Test Part 3. 

b. A block diagram. Fig. (5-7) is provided and table 
(5-6) is a list of equipment necessary to perform 
the test. 

Voice Communications System Test 

The Voice ystem Test is a test procedure whereby a 
systems ability to transfer intelligible information is 
measured by reference to an articulation Index (AI) . "AI 
has been shown to be a function of the link description (type 
.and frequency) , and the strength of the derived and interfer- 
ing signal, SRF and I, respectively; hence, for any given 
communications systems of such configuration that link and 
interferer descriptions and frequencies are known. AI Ccin 
be established as a function of SRF and I." [36, p. 1]. 

Values of AI have been determined in terms of audio S/N 
values permitting voice system's intelligibility to be 
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Sub-System Tost 1 "P" System 
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LIST OF TEST EQUIPMENT FOR SUB-SYSTEM TEST 1 

[31, p. 3-4] 



Description 

1. Magnetic Tape Deck 

2. Frequency Counter/Generator 

3. Spectrum Analyzer 

4. Attenuator 120 db 

5. Attenuator 12 db 

6. Micro voltmeter 

7. Dummy Load 

8. Power Divider 

9 . Test Tapes 

10 . Patch Cords 



Table (5-5) 
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Figure 5-7 ^iultichannel Test 



Sub-System Test 2. "P" System 
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LIST OF TEST EQUIPMENT FOR SUB-SYSTEM TEST 2 

[31, p. 3-9] 



Description 

1. Magnetic Tape Deck 

2. Test Tapes 



Table (5 - 6) 




I 



measured in terms of a S/N ratio. [37]. Articulation 
Index (AI) : A tape containing various audio frequencies and 

a precise amount of noise v/ill be introduced into the system 
and the receiver output will be measured with a noise 
measuring set to determine the intelligibility capability 
of the system. This test is only used in voice systems. 

[31 , p. iii ] . 

Procedure details for voice system test: 

a. A taped signal containing audio frequencies and 
noise is processed through the receiver and the 
output signal is measured by a noise measuring set. 

The noise measuring set looks at the noise passed by 
3 filters located in the voice band width. The 
difference between the input and output noise levels , 
expressed as S/N is equated to AI which is equated to 
intelligibility . 

b. This test procedure is used to determine the perfor- 
mance of a receiver configured for voice communications 
only. 

c. •• Figure 5-8 provides a block diagram of the test and 

table '5-7 provides a list of equipment necessary to 
perform the test. 

D. APPLICATIONS AND RESULTS 

Demonstrations have been conducted on board fleet units 
and at shore activities to validate design decisions made 
during SMT development, examine the universality of the 
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VOICE COmUNICATIONS TEST EQUIPI'lENT. [37] 



Description 

Random Noise Generator 
True RMS Voltmeter 
Amplifier 
Wave Analyzer 

Voice Interference Analysis System 

Digital Voltmeter 

Tape Recorder 

Noise Measuring Set 

Synthesizer 

Attenuator 

Attenuator/ Rot ary 



Table 5-7 
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procedures, and to introduce communications system users to 
the capabilities of SMT. Demonstrations have been conducted 
or on board USS En^lsnd, USS Preble, USS Intrepid, 
NAVCOMMSTA SFran , Naval Postgraduate School, Monterey; USS 
Constellation and other Commands. 

SMT performance has demonstrated that the original Long 
Range Goals can be satisfied. A number of present systems 
have been defined and described in the EDR. Numerical sys- 
tems perfourmance levels have been established in the RLMG 
and SMT has shown the techniques for measuring the systems 
p 0 jfformance levels. Several specific SMT objectives have 
also been satisfied. SMT tests are straightforward and can 
be understood by operating personnel at the fleet level 
after a short, formal course at Radiomen "B" School. To a 
large extent, existing test equipment can be utilized. 

Seven tests permit the entire communications system to be 
examined. Test sophistication is placed in the test equip- 
ment, not in its operation or in the technicians. SMT test 
sequence can be applied to any communications system, afloat 
or ashore, unaided. 

During December, 1972 the USS Constellation (CVA-64) was 
used for a demonstration and some of the test results are 
presented here. All communication systems available for 
operation v/ere tested. Several significant problems un- 
covered were corrected or brought to the operator's attention. 
Noteworthy are the facts that: 



68 



1 . 



The SMT disclosed severe maladjustments and failures 
not discernible by other test methods and 
2. The time taken to accomplish the tests was con- 
servatively less than five percent of that required 
by more conventional means available to the ship 
personnel . 

Problems encountered which could be corrected by adjust- 
ments and/or alignments were corrected. Problems requiring 
maintenance or repair were explained to cognizant, responsible 
personnel . 

It must be pointed out that the test sequence permitted 
testing of parameters not testable by other methods and that 
the results were able to point out pieces of equipment which 
were functioning but operating out of specifications. [38] 
Table 5-8 is an example of the SMT TTY test. After the 
page printers were checked, the alternate channel convertors 
for the "P" system (UCC-1 channels 1 to 16) were tested and 
6 out of 8 were found in need of maintenance. All "n" 
system channels (low. tones) were adjusted and their perfor- 
mance was greatly improved after the SMT tests pointed out 
weaknesses. A. total of 23 transmit and receive multicouplers 
were tested. Ten of the 23 were found to be out of specifi- 
cations. Twenty-one HF voice receivers were tested, operating 
in upper side band (USB) , and 4 were found in need of main- 
tenance. Twenty-one UHF voice receivers were tested and 12 
"fair" or "poor" performing units were found. Eighteen UHF 
transmitters were tested and 6 were operating within 
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SMT TTY TEST 



End equipment tests (TTY ' s and Page Printers) were per- 
formed 11 and 12 December 1972. Eight of the best operating 
machines were selected as reference standards for all other 
HF record circuit tests. 

All types of TTY distortion were introduced to each 
machine (eight at a time) in 5% increments, until the stress 
point of each machine was reached. A machine is considered 
to be in excellent operational condition if it can accept 40% 
distortion and print clean copy; 35% distortion input with 
clean copy is considered good; 30% fair; 25% marginal and 20% 
or below to be operationally in poor condition. 

Results of these tests were: 

Max. Distortion Max. Stress Type Distortion 
Machine with Clean Copy Point at Max. Stress Pt . 



LTP-C 


KYBD 


40% 


45% 


Space Bias 


LTP-C 


RCV 


35% 


40% 


Space Bias 


LTP-B 


KYBD 


40% 


45% 


Space Bias 


LTP-B 


RCV 


No test - Machine would 


not space 


LTP-PE 


KYBD 


45% 


45% 


1 hit on Mark Bias 


LTP-N4 


PP 


40% 


45% 


Space Bias 


LTP-PF 


KYBD 


40% 


45% 


Mark End 


LTP-N3 


PP 


35% 


40% 


Mark End 


LTP-PD 


KYBD 


35% 


40% 


Mark Bias 


LTP-A 


KYBD 


30% 


35% 


Mark Bias 


LTP-A 


RCV 


35% 


40% 


Mark Bias 


LTP-Nl 


PP 


40% 


45% 


All 


*LTP-PG 


KYBD 


35% 


40% 


Space Bias 


*LTP-N2 


PP 


35% 


40% 


Mark End 


LTP-N5 


PP 


35% 


40% 


Space Bias 


LTP-N7 


PP 


35% 


40% 


Space Bias 


LTP-TCl PP 


30% 


35% 


Mark Bias 


LTP-PB 


RCV 


40% 


45% 


Space Bias 


*LTP-PC 


KYBD 


35% 


40% 


Mark End 


LTP-PC 


RCV 


45% ■ 




1 hit on Space End 


LTP-PB 


KYBD 


35% 


40% 


Mark Bias 


Metro CH #8 


25% 


30% 


Mark Bias 


*NOTE 


LTP-PG 


KYBD 


Margin adjustment 


required - 


3 characters short 


LTP-N2 


PP 


Spacing and line 


feed problem 




LTP-PC 


KYBD 


Margin adjustment 
[38] 


required - 


2 characters short 



Table (5-8) 
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specifications . Seven HF transmitters were tested and none 
were found to be operating within specifications. [38] 

E. PROBLEMS WITH PRESENT PROGRAIi AND AREAS FOR FURTHER STUDY 

1. The SMT program has completed some system and siob- 
system feasibility studies and needs to devote additional 
attention to its final two levels of testing, i.e., equipment 
and circuits. 

2. Once laboratory procedures and instrioment ation re- 

quirements have been established alternate, suitable, test 
equipment must be examined and a trade-off analysis conducted 
between equipment cost and equipment capability in order to 
reduce the price of each set of test equipment from $48,000 
to the stated goal of $5,000 per set, while still satisfying 
the mission of SMT. The ideal cost effective solution, and 
one which certainly is feasible is to determine_ what pieces' 
of test equipment, presently in the Naval inventory, could be 
used to perform the SMT test and thej^eb y not incur any addi- 
tional equipment exp ense . The goal should be: that by 

providing a trained radioman the tapes , ^ could conduct the 
test at no additional expense to the Navy other than the cost 
of his training and the tapes. 

3. If the fleet is to be the ultimate user of SMT, the 

fleet must have trained personnel available which implies a ' 

~^r~ S/ 

t rai ning pro^nam,. s yllabus and instructors, etc. These needs 
must be planned for as long lead time items so they can be 
available when SMT is vailable for fleet use. 
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4. One of the goals of the SMT program is to replace 
several less efficient tests presently being used. These 
tests and/or procedures which are no longer needed must be 
identified and removed from the inventory. The SMT program 
should be able to pay for itself with tests , equipment and 
training it replaces while at the same time it enhances the 
tests which it complements. 

5. The SMT program must keep in mind its long range 
goals and specific objectives and continue to devote its 
attention in these areas . Work with the fleet should be 
secondary to these goals and objectives until the total 
package is prepared. By dividing its attention between pro- 
gram development and fleet applications it v;ill most likely 
result in a less efficient product and it will extend the 
development phase longer than would be otherwise necessary. 
These detractions must be weighed against possible benefits 
of present fleet utilization. 

6. When Naval Electronic Systems Command has completed 
the SMT development program, it must have prepared an imple- 
mentation plan or OPORDER to introduce its technology to the 
fleet. The transfer of technology, to be effective and fully 
utilized, must be prepared, programmed and conducted in a 
well planned and executed maneuver. 

7. Once SMT testing procedures for communications 
System, Sub-system, equipment and circuits have been designed, 
an adequate level of testing at each level must be developed. 
The depth, intensity and frequency of tests for components 
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and systems must be determined. The areas of degraded mode 
(total systems degradation or component failure which re- 
quires use of an alternate procedure) and systems sensitivity 
to atmospheric or operational phenomena must be determined. 

8. Studies must be conducted to determine the time period 
a calibrated (SMT tested) system remains in calibration. In 
order to ensure optimum systems availability should the sys- 
tem be tested daily, weekly, once a quarter, etc.? Once 
this "validity interval" is known an implementation program 
must be drawn up which would select either or both of the 
following courses of action: 

1) If the validity interval is determined to be measur- 
able in extended periods of time (month or more) the 
SMT test sequence could be administered to a ship as 
part of its regular overhaul, pre-deployment ORI , or 
during every in-port period. This approach would 
dictate mobile teams stationed in key domestic and 
foreign ports. 

2) If the validity interval is measurable in hours, days 
or weeks it may be desirable to have each major 
operating unit able to test itself on a frequent 
schedule. This approach would perm it a ship to con- 
tiguously monitor its communication_ system and acjiieve 
maximiam availability. This solution would be expen- p 
sive in terms of manpower, equipment and training. 

A trade-off analysis should be conducted between the 
resultant availability and cost. 
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be determined that the optimum solution is a blend 
of the two approaches with major ships carrying its own SMT 
equipment v/hile mobile teams provide periodic readiness in- 
spections. Once either or both alternatives is selected, 
implementation plans must be prepared and funding sought to 
accomplish the established goal. 

9. Voice measuring systems must be refined. The present 
procedure calls for looking at three 100 Hz windows in the 
voice band from 200 to 6100 Hz and correlating the data to 
determine intelligibility. Studies must be continued to 
determine the optimum location of the three windows emd to 
refine the procedure. 

10. Studies must be conducted to determine the signal 
reproduction loss due to multipath reception and the effect 
of quad diversity upon the UCC-1. 

11. Cross talk between systems and components must be 
studied so that its effects can be predicted, tested and 
corrected . 

12. Studies of compatibility of the SMT procedures and 
Secure systems must be conducted. Until Secure systems can 
be incorporated into communications systems test sequence the 
sequence will not be able to realize its full potential. 

13. The validity of the atmospheric test must be in- 
creased. This can be accomplished by improving the atmo- 
spheric simulation techniques. 
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VI. POTENTIAL BENEFITS AND APPLICATIONS OF SMT 



The potential applications and benefits to be derived 
from applications of the SMT concept are niimerous. The ben- 
efits include increased standardization, more definitive 
specifications for procurement, reduction of communication 
systems duplication, reduced manpower needed to support 
communications, increased availability and a means to mea- 
sure the degraded level at which a system is operating. 
Numerous potential applications are also listed. 

A. POTENTIAL BENEFITS OF SMT 

Standardization is one of the basic principles upon which 
the SMT concept was founded and it is the key to tlie high 
degree of success that has been achieved during the initial 
phase of development. The standardization principles employed 
in the development of the SMT concept should also be applied 
to the. development of new equipment by Navy laboratories and 
commercial contractors to ensure that the same techniques are 
used to establish performance standards. Standards of per- 
formance , based on SMT requirements , should be an integral 
part of a system, sub-system, equipment and component de- 
sign specifications. All equipment procurred, as a result of 
a production contract, should comply with the performance 
criteria established by SMT as the minimum acceptable for use 
in an operational environment. 
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As a result of the use of SMT procedures in evaluating 
systems performance, standardization of test procedures would 
be universal regardless of the system configuration or site 
location. Any manager, operator, or supplier should be aware 
of the benefits of standardization. 

When the operators and industry engineers select a common 
unit of measurement, as the EDR, RLMG and SMT procedures can 
provide for communications, everyone will speak the same 
language. Definitive specifications can be stated, and under- 
stood and construction can be monitored in terms of quantified 
parameters, not qualified criteria as is often the practice at 
present. The operator can plan on achieving and holding a 
performance level, which will permit him to make meaningful 
judgments as to his needs and capabilities. 

Once a common unit of measurement has been selected all 
communications systems can be compared and contrasted with 
each other. It is quite certain that duplication of communi- 
cation systems exists between members of the Armed Forces , 
nximerous other government agencies, and within industry. 
Duplication is taken to mean two systems which perform the 
same task or similar tasks and for which research and develop- 
ment costs, learning curve costs and implementation costs 
were duplicated. If all concerned agencies had used one mea- 
surement scale it would have been possible to prevent the 
needless additional expense of developing and/or building 
and/or introducing the same system twice. 
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With a test procedure such as SMT, communications systems 
manpower requirements could be reduced in several ways. 

Since an entire external communications system can be checked 
in 5% to 20% of the time needed by present test procedures, 
less personnel are needed to perform the tests. Less down 
time is needed to perform maintenance. Systems and components 
can be tested while installed which means equipment does not 
have to be removed and replaced and it is not exposed to 
damage while handling. Additionally, only defective com- 
ponents need be removed and a better indication of the failure 
can be provided greatly easing the maintenance requirement 
which means less personnel are needed to perform overhaul or 
depot level maintenance. 

The SMT testing procedures will enable the user to deter- 
mine the level of performance of his system. The user will 
be aware of any degradation of the system's performance. He 
will be able to measure the level of degradation and make a 
decision, based upon quantitative infoimation, regarding his 
ability to complete an assigned mission. 

Proper utilization of the SMT techniques will increase 
the communications systems availability. Since availability 
can be achieved through maintainability and maintainability 
needs an adequate test procedure and SMT is, not only, an 
adequate test procedure , but an improvement over einy other 
available test procedure, system availability will increase 
which means the system operator and user will derive a benefit 
from utilization of the Standard Measurement Technique. Since 
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SMT measures degraded mode operations , maintenance can be 
planned in anticipation of a catastrophic failure which means 
the system should not, as often, suddenly be put out of 
commission. When catastrophic failure does occur, SMT will 
permit quicker trouble shooting and permit a faster restoration 
of the failed system or component. 

Numerous additional benefits" may be derived by implemen- 
tation of SMT. When a garbled message is received, the 
receiver could test his system to ensure it is operating 
within specifications. He could inform the sender of the 
garbled message and the sender could do likewise. It may 
even be advantageous to have the SMT test signal broadcast 
from one imit to another in order to test the communications 
system on a macro level . 

An individual unit could introduce a series of messages 
into its own receiver system and then observe how the messages 
are handled by its local message distribution system. 

Taking this idea one step further could enable SMT to 
make its greatest contribution to Naval Communications . Since 
SMT can measure the level of performance of a communications 
system, as on a ship, and assign a quantitative value to it, 
it could assign a quantitative value to a total ship's 
communications suit by testing all on-board communications 
systems and summing their level of performance. A method of 
weighting the various systems would have to be devised. In 
this m^mner it could be stated that ship X can communicate 
with a; Y (30%), level of distortion with a Z; (95%) , confidence 
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interval. Once this is done every ship in the Navy could have 
a quantitative value (s) assigned to its communications suit. 
Weak performers could receive more attention and strong per- 
formers checked to see what beneficial practices could be 
utilized by other vessels. The same concept could be applied 
to all Naval Communications Stations whereby their ability to 
communicate could be quantified. Once ships and stations 
levels of performance have been determined the test could be 
applied from the communications stations to the ship and vice 
versa by transmitting the test tape. This step would enable 
the performance level of the trunk to be quantified. 

Up to this point we have addressed equipment orientated 
applications. It is possible to include personnel, message 
handling, distribution and management. By using test tapes 
with a pre-determined sequence of messages introduced at the 
communications station, the flag ship, or at an individual 
ship's receiver, the ability of a communications system to 
put a message in the hands of the ultimate, intended user 
could be evaluated. This tape could include messages dealing 
with operations, administration, supply, etc., and range from 
FLASH to ROUTINE in priority. In this memner, all aspects 
of the total communications system, i.e., equipment, personnel, 
management, etc., could be tested and weak points identified. 
The test could be repeated at any location or unit and permit 
a valid, equal comparison between two or more communications 
systems . 
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Test tapes could be transmitted via satellite in order 
to determine the level of performance at which various 
satellite channels are operating. 

Since there is no major communications system test avail- 
able/ SMT would complement present sub-system and component 
tests. Some tests will be redundant and no longer needed and 
they can be removed from the inventory producing a cost 
savings. However the two primary benefits of SMT must not be 
overshadov/ed. First it is a means to check a communications 
system, not components, and secondly it provides a means to 
measure degraded perfoinnance . Since very few systems are 
operating at full performance levels, an ability to measure 
the degraded level of performance is necessary. 

Retrofit installation of a system and/or equipment is 
an expensive process. The early acceptance of SMT as a 
standard installation for new construction could result in a 
considerable savings. As an interim measure, consideration 
should be given to the installation of sensors required for 
SMT in nev7 construction pending a decision to include SMT in 
initial outfitting. 

B. POTENTIAL APPLICATIONS OF SMT 

The potential application of Standard Measurement Tech- 
niques (SMT) to electronic systems, other than afloat/ashore 
communication systems, is based on the documented successful 
achievement of initial objectives obtained in the development 
of SMT as an integral part of the Navy's overall System 
Performance Measurement Program. 
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The application of the SMT engineering concept to the 
communication links of Tactical Data Systems such as NTDS , 
ATDS, and MTDS , is a natural extension of the initial appli- 
cation of SMT. Testing hardware in TDS equipped aircraft 
(E2C, P3C, F14) using real world, simulated inputs, and em- 
ploying SMT procedures for determining the performance of 
data links prior to being airborne, would be valuable. The 
SMT concept could also be employed while airborne to monitor 
data link performance, determine source of data link problems 
(own aircraft or termination station) , and isolate equipment 
that is operating below established performance levels . 

The techniques of SMT are universal in application to 
electronic systems. Although, the initial application is 
for testing naval communication systems, SMT is applicable to 

a. Tactical Data Systems 

b. Avionic Systems 

c. ASW Systems 

d. Weapons Systems 

e. Interior Communication Systems (ICS) 

f. Command and Control Systems 

g. Navigation Systems 

h. Aircraft Voice Systems 

i. Submarine Communications Systems 

j . ECM Systems 

Plans have been drav;n up to "Hard Wire" the SMT system 
into the communications system of the USS NIMITZ (CVAN-68) 

[23] and into the Patrol Frigate (PF) class which is in the 
development stage [39]. 
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VII. CONCLUSION 



The need for a Naval Communications System Level Test 
Procedure exists. Until this need is satisfied, personnel 
operating the system will not know or be able to use all the 
potential of the system which they are working with. Once 
the level of performance of a system can be quantified, using 
meaningful criteria, such as, the amount of degradation which 
it can accept and still be usable for communications or what 
percentage of the system is operating above a predetermined 
level, the managers of the system can determine how much 
through-put the system can provide. This knowledge, capa- 
bility and availability, is the first step towards efficient 
utilization of present and future resources. A test pro- 
cedure is needed which can provide this knowledge. 

It has been shown that all existing test procedures are 
inadequate and mislabeled. Virtually all are called systems 
level test procedures when they are really subsystem level 
procedures measuring' output power, percentage modulation, 
etc. 

A system level test procedure should be able to accomplish 
the following: 

1) Quantify the level of performance of the system or 
determine the present level of degradation at which the sys- 
tem is operating. 

2) Determine how much degradation in the input signal a 
system can accept and still put out an error-free or acceptable 



copy. 
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3) Determine the communications system ability to 
communicate; its reason for being. 

The Standard Measurement Technique (SMT) , presently under 
development by the Naval Electronic Systems Command, Southwest 
Division, is a system level test which can fulfill this need. 
By inputing signals of known degradation and quantifying the 
output it can measure the ability of a system to pass infor- 
mation. However, SMT is not an end-all. It does not provide 
all the information which a manager needs in order to operate 
his system. It should be used to compliment some present sub- 
systems level procedures while others should be deemphasized . 
SMT can be an excellent tool in the hands of the operators 
and V7hen used in conjunction with QMS can provide a complete, 
instantaneous systems status report. It can be used to 
point out where the maintenance effort should be applied. 
PMS/3M and Pomsee should be applied as maintenance procedures 
to failed components located by SMT and QMS. 

SMT development, by Naval Electronic Systems Command, 
must be completed. Alternate cost effective pieces of test 
equipment must be found. Test procedures must be firmed and 
alternative applications must be examined. The SMT validity 
interval needs to be determined so that a decision can be made 
which will evaluate the costs and benefits of applying SMT 
to operational units on a periodic schedule, such as once a 
quarter or before a cruse, or having it built into all units 
so it can be used continuously. 
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The Standard Measurement Technique has the potential to 
deliver system level testing and provide Naval Communications 
with a complete, true, system, integrated test structure. 



I 
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APPENDIX A 



Major Test Equipment Functions 

The functions of the major test equipment used in the 
"P" system test are as follows: 

a. Magnetic tape deck and interface buffer used to in- 
put prerecorded standard test message tapes. 

b. Spectrum Analyzer is used to measure the individual 
channel levels of a composite tone, i.e., distortion 
levels, inter-modulation. 

c. Dummy load to absorb the full power transmit signal 
to acceptable testing levels. 

d. Attenuators used to reduce the receiver input voltage 
levels to acceptable testing levels . 

e. Micro-voltmeter used to measure voltage level at 
receiver . 
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